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ABSTRACT 

For  most  standards  bodies,  the  validation  and  maturation  process  is  dependent  on 
motivated  members  of  the  standards  community  to  develop  reference  systems  or  components  and 
to  provide  the  governing  body  with  the  necessary  data  and  details  to  support  maturing  a  given 
specification  or  set  of  specifications.  Although  this  has  worked  well  for  other  standards  bodies,  the 
VICTORY  Standards  Support  Office  (VSSO)  recognized  early  that  validation  would  be  key  in 
rapidly  defining  usable  specifications  for  the  Army  ground  vehicle  community.  Understanding  the 
importance  of  validating  specifications,  the  VSSO  formally  defined  a  validation  process  that  is 
used  to  aid  in  maturing  the  VICTORY  Standard  Specifications.  This  paper  will  focus  on  explaining 
the  formalized  validation  process  that  is  applied  to  the  VICTORY  Standard  Specifications. 


INTRODUCTION 

Validation,  for  the  Vehicular  Integration  for 
C4ISR/EW  Interoperability  (VICTORY)  initiative, 
evaluates  specifications  generated  by  VICTORY 
working  groups  and  implements  reference  functional 
components  when  applicable.  Most  standards  bodies 
rely  on  their  membership  to  provide  validation 
support,  which  is  an  integral  part  of  the  VICTORY 
Standards  development  process.  The  VICTORY 
Standards  Support  Office  (VSSO)  formalized  the 
validation  process  to  enable  rapid  maturation  of  the 
VICTORY  specifications.  The  validation  process 
focuses  on  showing  that  1)  the  specifications  are 
complete  and  are  not  ambiguous  2)  when 
implemented  and  integrated  in  a  small  scale  system, 
the  component  and  interface  specifications  results  in 
the  functional  performance  that  was  expected  3)  the 
performance  such  as  latency,  network  utilization,  and 
general  responsiveness  is  known  and  documented,  4) 
the  development  effort  and  the  hardware  required  for 
an  implementation  of  the  components  and  interfaces 
used  in  the  experiment  are  known  and  documented 


and  5)  when  implemented  and  integrated  by 
independent  entities,  the  resulting  interoperability  is 
known,  documented,  and  is  deemed  acceptable. 

Prior  to  delving  into  the  intricacies  of  VICTORY 
Validation,  it  is  necessary  to  gain  a  high-level 
understanding  of  the  VICTORY  process  for  the 
creation  and  validation  of  the  specifications.  Figure  1 
is  representative  of  this  process. 


Figure  1:  High-Level  Specification  Process 
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The  first  step  in  creating  a  specification  requires  a 
working  group  to  evaluate  the  VICTORY 
Architecture  to  determine  which  interfaces  and 
components  need  to  be  specified  and  assign  a 
specification  priority  to  each.  This  priority  dictates 
the  order  in  which  the  topics  will  be  presented  to  the 
working  groups  but  does  not  guarantee  the  order  in 
which  a  given  specification  will  become  available. 
With  initiation  from  the  members  the  leadership  for  a 
given  working  group  devises  an  initial  Change 
Proposal  (CP)  plan  and  presents  it  to  the  working 
group  membership.  Once  the  working  group 
membership  has  agreed  on  the  content,  it  will  be 
documented  and  integrated  into  the  VICTORY 
Standard  Specifications  with  an  experimental 
maturity  level.  Figure  2  below  shows  the  maturity 
levels  as  defined  by  the  VSSO  and  how  they  map  to 
the  types  of  validation  mentioned  previously. 


Initial 

Validation 

-  Preliminary  Specification 

-  Informational  Specification 

Additional 

-  Experimental  Specification  validation 

-  Proposed  Standard  Specification  J  Depioyme 

-  Draft  Standard  Specification 

-  Standard  Specification 

Figure  2:  VICTORY  Maturity  Levels 

Once  an  experimental  specification  has  been 
documented,  it  will  enter  the  validation  process.  The 
validation  process  has  been  formalized  to  produce 
more  objective,  reproducible  and  thorough 
experiment  results  and  consists  of  two  validation 
tracks  as  shown  in  Figure  3. 


Recaks  A  MU  VWfetfaa 


Figure  3:  High-Level  Validation  Process 


The  first  track  is  referred  to  as  Initial  Validation 
and  it  is  responsible  for  performing  experiments  on 
the  specifications  that  are  at  an  experimental  maturity 
level.  This  Initial  Validation  effort  is  led  and  funded 
by  the  VSSO.  Initial  Validation  requires  creating  an 


experiment  plan,  performing  the  experiment,  and 
reporting  the  results  and  recommendations  back  to 
the  working  group.  After  review,  the  working  group 
makes  the  final  recommendation  to  the  VSSO  to 
mature  a  specification  to  the  proposed  level.  Once 
matured  to  the  proposed  level,  the  specification  is 
inserted  into  the  second  validation  track,  referred  to 
as  Additional  Validation.  Additional  Validation  is 
performed  by  independent  entities  that  do  not  fall 
within  the  VSSO  footprint.  It  follows  the  same 
procedures  as  Initial  Validation;  however,  Additional 
Validation  experiments  also  include  interoperability 
evaluations.  Based  on  a  review  of  the  results  and 
recommendations  from  the  Additional  Validation 
experiments,  the  working  group  decides  if  a 
specification  is  ready  for  the  next  maturation  level.  If 
so,  the  recommendation  is  made  to  the  VSSO 
requesting  that  the  specification  become  a  Draft 
Standard  Specification.  The  remainder  of  this  paper 
will  focus  on  describing  the  formalized  process  for 
both  Initial  and  Additional  Validation  for  the 
VICTORY  Standard  Specifications  and  will  provide 
a  brief  status  of  what  the  validation  process  has 
accomplished  thus  far. 

INITIAL  VALIDATION 

Initial  Validation,  is  the  first  step  in  maturing  the 
VICTORY  Standard  Specifications  and  provides  the 
Army  ground  vehicle  community  with  a  sense  of 
confidence  that  the  specifications  can  be  realized  in 
fully  functional  components  and  are  useable  in  real 
world  systems.  One  might  wonder  how  does  initial 
validation  provide  this  level  of  confidence  to  the 
community?  Consider  the  artifacts  from  the  initial 
validation  experiments.  For  most  specifications,  the 
initial  validation  experiment  requires  a  software 
reference  implementation  of  the  intended  interfaces 
to  be  evaluated.  Throughout  the  course  of  the  initial 
validation  effort,  these  reference  components  have 
been  integrated  into  an  extensive  library  of  reference 
functional  components  that  can  be  referred  to  during 
the  development  of  production  systems.  The 
reference  components  show  that  an  associated 
specification  is  usable,  implementable,  functionally 
correct,  complete  and  non-ambiguous.  Experiments, 
in  many  cases,  document  metrics  about  functional 
performance  and  resource  requirements.  This  data 
can  be  valuable  when  one  needs  to  know  what  the 
system  performance  for  a  given  component  is  likely 
to  be.  The  remainder  of  this  section  will  outline  the 
process  for  performing  initial  validation  experiments 
and  provide  a  brief  status  overview  of  the  progress  to 
date  as  well  as  what  is  planned  for  initial  validation 
in  the  upcoming  scheduled  releases  for  the 
VICTORY  Standard  Specifications.  The  initial 
validation  process  requires  high-level  planning  to 
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determine  what  specifications  will  be  entering  the 
initial  validation  track,  developing  initial  validation 
experiment  plans  specific  to  the  specifications  being 
evaluated  and  conducting  the  experiment  defined  by 
the  plan. 

The  high-level  planning  is  the  most  crucial  aspect 
of  the  initial  validation  track.  It  requires  a  mapping  of 
projected  capabilities  to  one  or  more  change 
proposals  (CPs)  that  each  working  group  deems 
necessary  to  describe  the  specifications  that  will 
produce  a  capability.  A  VICTORY  Capability  is 
comprised  of  one  or  more  specifications  that  have 
been  matured  to,  at  least,  the  “Proposed”  level.  The 
CP  plan,  developed  by  the  working  groups,  provides 
a  target  date  for  defining  an  experimental  level 
specification  and  is  indicative  of  the  earliest  date  that 
an  experiment  can  start.  In  other  words,  the  resulting 
dates  provided  in  the  Change  Proposal  plan 
determine  when  initial  validation  experiments  will  be 
scheduled  in  the  Initial  Validation  plan. 

Initial  validation  experiments  are  the  thoroughfare 
for  maturing  the  VICTORY  Standard  Specifications 
to  the  “Proposed”  level.  There  are  three  main  goals  of 
initial  validation:  1)  ensure  each  specification  is  clear 
and  complete;  2)  develop  reference  functional 
components  of  each  specification  when  applicable; 
and  3)  mature  the  specifications  to  the  “Proposed” 
level.  There  are  two  possible  methods  for  performing 
initial  validation  experiments  for  the  VICTORY 
Standard  Specifications.  One  method  is  to  perform 
entirely  theoretical  experiment  and  is  commonly  used 
when  necessary  hardware  is  unavailable  or  when  a 
particular  capability  is  readily  available  for  a  given 
specification  (i.e.  3rd  party  software  tools  such  as  an 
Ethernet  management  interface  for  voice  radios). 
During  a  theoretical  experiment,  VICTORY  Standard 
Specifications  will  be  scrutinized  for  clarity  and 
completeness.  Upon  completion  of  the  evaluation  of 
the  specifications,  a  detailed  written  analysis  of  all 
specified  functionality  will  be  performed.  The  second 
and  the  preferred  method,  shown  in  Figure  4,  will  be 
used  to  describe  the  typical  process  of  an  initial 
validation  experiment. 


Figure  4:  Initial  Validation  Experiment  Process 


The  preferred  method,  for  performing  initial 
validation  experiments,  consists  of  multiple 
independent  evaluations  of  the  specifications.  The 
evaluators  for  the  experiment  include  an  Experiment 
Designer  and  one  or  more  Software  Developers.  An 
Experiment  Designer  evaluates  the  specifications 
with  the  focus  being  how  to  develop  experiment 
procedures  that  will  allow  every  aspect  of  the 
specification(s)  to  be  evaluated  through  the  use  of  a 
representative  functional  component.  In  parallel,  the 
Software  Developer  evaluates  the  specification(s) 
with  a  focus  on  designing  and  implementing  one  or 
more  representative  functional  components.  During 
the  evaluation  of  the  specifications,  the  Experiment 
Designer  and  Software  Developer  perform  their 
evaluations  completely  independent  of  one  another 
and  record  any  issues  discovered.  During  this  stage 
of  an  experiment,  the  two  evaluators  are  separated  by 
a  “firewall”  thereby  allowing  multiple  independent 
evaluations  to  be  achieved. 

Upon  completing  the  evaluation  process,  the 
Experiment  Designer  writes  an  initial  validation 
experiment  plan  while  the  Software  Developer 
designs  and  develops  the  representative  functional 
component(s).  The  initial  validation  experiment  plan 
states  the  goals  of  the  experiment;  serves  as  a  record 
of  specifications  being  evaluated;  provides  detailed 
procedures  for  evaluating  the  specifications;  records 
any  discrepancy  found  during  the  evaluation;  and 
provides  recommendations  back  to  the  working 
groups  with  regard  to  maturing  the  specifications  and 
addressing  any  discrepancies.  The  representative 
functional  components  are  used  for  evaluation 
purposes  during  the  experiment  procedures  and  will 
ultimately  serve  as  reference  implementations  for  the 
VICTORY  community.  Upon  completion  of  the 
initial  validation  experiment  plan  and  development  of 
the  representative  functional  components,  the 
Experiment  Designer  and  Software  Developer(s)  will 
combine  their  efforts  to  setup  a  representative 
experiment  system  and  evaluate  the  procedures 
outlined  in  the  experiment  plan. 

Validation  Experiment  Plan 

An  experiment  plan  outlines  the  procedures  for 
conducting  an  initial  validation  experiment  for  a 
single  specification  or  group  of  specifications.  After  a 
working  group  documents  a  given  CP,  an  Experiment 
Plan  can  be  designed.  An  Experiment  Plan  is 
designed  to  validate  whether  or  not  the  specifications, 
documented  by  a  Working  Group,  are  un- ambiguous 
and  whether  there  is  enough  content  to  enable  the 
development  of  a  representative  functional 
component.  In  an  effort  to  streamline  designing  the 
Experiment  Plans,  a  template  is  used.  This  template 
provides  an  outline  for  how  an  experiment  shall  be 
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designed  and  provides  uniformity  across  all  initial 
validation  experiments. 

The  procedure  for  designing  an  Experiment  Plan 
consist  of  the  following  phases: 

1.  Review  of  the  documentation  provided  by 
the  working  group.  The  documents  consist  of  the 
specification  document,  schema,  and  Web  Service 
Description  Language  (WSDL)  files.  During  this 
phase,  the  content  is  scrutinized  for  ambiguities. 

2.  Develop  experiment  procedures  for 
validating  the  documented  specifications.  The 
procedures  target  each  specification  being  evaluated. 
Although  they  vary  based  on  the  specification  being 
evaluated,  it  is  common  for  them  to  evaluate  latency, 
functional  behavior,  and  bounds  checking.  In  most 
instances,  an  experiment  will  require  representative 
functional  software  components  to  be  developed  to 
evaluate  the  procedures  and  to  show  that  the 
specifications  can  be  implemented. 

3.  Create  a  logical  and  physical  design  for 
executing  the  experiment.  This  phase  designs  the 
hardware  and  software  configurations  necessary  to 
perform  the  experiment. 

Software  Development 

An  integral  part  of  initial  validation,  software 
development  evaluates  whether  or  not  the  VICTORY 
specifications  can  be  transformed  into  a  functional 
component.  This  portion  of  Initial  Validation  focuses 
on  implementing  a  functional  software  component,  or 
components,  based  on  the  specification  document 
content,  schema  and  WSDLs.  To  streamline  the 
software  development  process  for  validation 
experiments,  a  software  framework  has  been 
developed  and  is  composed  of  common  software 
component  libraries  that  have  been  developed  in  prior 
validation  experiments.  For  each  validation 
experiment  conducted,  all  software  components  or 
libraries  deemed  reusable  are  added  to  the  software 
framework  used  for  validation  experiments. 

The  phases  for  developing  initial  validation 
software  components  are  as  follows: 

1.  Review  the  specification  document,  schema 
and  WSDLs  to  determine  the  functionality  that  a 
software  component  shall  provide.  During  this 
review,  the  schema  and  WSDLs  are  evaluated  for 
ambiguities  and  crosschecked  with  the  specification 
document  to  ensure  consistency. 

2.  Develop  a  Software  Design.  During  the 
design  process  for  the  software,  the  designer 
determines  which  software  components  are  necessary 
to  implement  a  functional  component  based  on  the 
review  of  the  specification  document,  schema,  and 
WSDLs.  It  is  also  determined  if  existing  libraries 
from  previous  experiments  can  be  used  to  streamline 
the  development  process.  In  most  cases,  there  are  a 


number  of  libraries  that  will  be  reused  to  perform 
routine  functions  that  will  not  have  an  affect  on 
results  of  an  experiment. 

3.  Develop  Software  Components.  During  this 
phase,  the  software  development  team  will  develop 
the  software  components  as  outlined  by  the  software 
design. 

Performing  Experiments 

After  developing  the  experiment  plan  and  reference 
component  implementations,  the  VICTORY  Initial 
Validation  Facility  (VIVF)  is  configured  in 
accordance  with  the  logical  and  physical  designs 
outlined  in  the  experiment  plan.  The  software 
components  used  typically  consist  of  open  source 
applications  and  those  developed  in  accordance  with 
the  specifications  being  evaluated.  When  performing 
an  experiment,  the  evaluator  documents  all 
discovered  ambiguities,  inconsistencies  and  flaws 
that  may  arise.  These  items  are  referred  to  as 
findings.  The  experiment  plan  contains  a  section  to 
record  the  results  of  the  experiment,  all  findings  that 
have  been  uncovered,  and  recommendations  for 
resolving  the  findings.  Recommendations  for 
maturing  the  specifications  that  have  been  evaluated 
are  also  documented  in  the  experiment  plan.  All 
results,  findings  and  recommendations  are  reported 
back  to  the  working  groups  in  the  form  of  a  CP.  The 
CP  is  used  to  allow  the  working  groups  to  determine 
how  the  specifications  are  affected  as  a  result  of  the 
experiment. 

Depending  on  the  actions  taken  by  the  working 
groups,  the  experiment  plan  and  supporting  software 
may  need  to  be  revised.  In  cases  where  the 
supporting  software  cannot  be  developed  in 
accordance  with  a  documented  specification,  the 
initial  validation  experiment  team  will  make  a 
recommendation  for  how  the  specification  could  be 
documented  and  proceed  with  implementation  based 
on  that  recommendation.  If  a  working  group  decides 
to  reject  a  recommendation  and  seek  alternative 
solutions  for  a  specification,  the  experiment  plan  and 
supporting  software  will  need  to  be  revised  and  the 
experiment  will  need  to  be  performed  again.  In 
extreme  cases,  an  entirely  new  experiment  may  be 
required. 

Initial  Validation  Status 

Table  1  shows  how  many  experiments  were 
completed  and  the  number  of  proposed  standards  for 
each  version  of  the  VICTORY  Standard 
Specifications  that  have  been  released. 
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Specification 

Version 

Number  of 
Experiments 

Number  of 
Proposed 
Specifications 

1.0 

20 

108 

1.1 

14 

35 

1.2 

16 

37 

TOTAL 

50 

180 

Table  1:  Initial  Validation  Progress 


Table  2  shows  how  many  experiments  are  planned 
and  the  approximate  number  of  proposed  standards 
that  will  be  achieved  as  a  result  of  the  1.3  and  1.4 
releases  of  the  VICTORY  Standard  Specifications.  It 
should  be  noted  that  the  data  in  Table  1  and  Table  2 
was  based  on  data  that  was  current  when  this  paper 
was  written.  Since  writing  this  paper,  version  1.3  of 
the  VICTORY  Standards  Specification  has  been 
released. 


Specification 

Version 

Number 
of  Experiments 

Number  of 
Proposed 
Specifications 

1.3 

~12 

~20 

1.4 

~18 

~40 

TOTAL 

~30 

~60 

Table  2:  Initial  Validation  Plans 


INDEPENDENT  DEVLOPMENT  AND 
VALIDATION 

US  Army  Tank  Automotive  Research, 
Development  and  Engineering  Center  (TARDEC)  - 
Vehicle  Electronics  and  Architecture  (VEA)  group 
developed  an  independent  implementation  [3]  of 
VICTORY  Architecture  and  Services  at  their 
VICTORY  SIL  with  the  purpose  of  maturing  and 
standardizing  ground  vehicle  electronic  architecture, 
sub-system  interfaces  and  compliance  testing.  An 
additional  set  of  experiments  were  performed  at  the 
SIL  to  provide  verification  and  validation  of  the 
VICTORY  1.0  Architecture  core  services  and  data 
bus  standards.  The  testing  implied  that  when 
implemented  correctly  with  the  standards  format 
specified  by  the  VICTORY  1.0  will  provide 
VICTORY  services  data  to  the  VDB  (Victory  Data 
Bus)  as  shown  in  Figure  5. 


Figure  5:  VICTORY  Services  network  as 
implemented  in  the  VICTORY  SIL 


The  experiments  conducted  evaluated  the  interface 
specifications  by  integrating  software  clients  and 
services  developed  using  the  specifications,  and 
evaluating  the  resulting  functional  behavior  and 
performance.  The  TARDEC  Vehicle  Electronics  and 
Architecture  (VEA)  group  executed  this  set  of 
additional  validation  experiments,  utilizing  their 
VICTORY  System  Integration  Laboratory  (SIL). 


Additional  Validation  Process 

The  test  plans  and  the  experiments  performed  are 
similar  to  those  of  the  VIVF,  the  results  and  any 
additional  findings  are  recorded  in  a  series  of  test 
reports. 

Additionally  two  test  tools  were  used  for  managing 
and  monitoring  the  VDM’s: 

a)  Wireshark  VDB  plug-in 

A  custom  dissector  plug-in  for  Wireshark  version 
1.2.8  is  developed  for  the  VICTORY  SIL  and  is  used 
as  a  tool  for  testing  and  monitoring  VDM’s.  This 
dissector  captures  UDP  VICTORY  Data  Messages 
(VDMs)  and  breaks  them  down  into  their  specific 
header  and  data  fields.  It  also  provides  a  filter  to 
look  for  VDM  messages  and  the  ability  to  log 
captured  VDMs  to  a  formatted  text  file. 

b)  Terminal  &  GUI  Client’s  for  DoT  & 
Orientation  VDM  Management 

Two  clients  were  developed  to  manage  the  VDM’s: 
One  is  a  command  line  client  and  the  other  is  a  GUI 
based  client  as  shown  in  Figure  6.  Both  of  these 
clients  perform  the  same  VDM  control  functionality, 
i.e.  to  enable/disable,  set  data  rates,  set  update  period, 
etc. 
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Figure  6:  Terminal  and  GUI  Client  for  VDM 
Management 


Additional  Validation  Status 

The  Table  3  below  shows  the  number  of  planned  to 
completed  experiments  at  47%  towards  verification 
of  the  1.0  standards.  All  the  testing  is  performed  at 
the  TARDEC  VICTORY  SIL.  As  the  TARDEC 
VICTORY  SIL  adapts  other  versions  of  the 
VICTORY  standard  specification,  additional 
validation  experiments  will  be  planned  and 
completed. 


Specification 

Version 

Number  of 
Proposed 
Specifications 
Tested  and 
Verified 

Number  of 
Proposed 
Specifications 

1.0 

45 

96 

TOTAL 

45 

96 

Table  3:  Additional  Validation  Progress 
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